Self-slimmed Vision Transformer

Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. However, such powerful bring a huge computation burden, because of exhausting token-to-token comparison. The previous works focus dropping insignificant tokens to reduce computational cost ViTs. But when ratio increases, this hard manner will inevitably discard vital tokens, which limits its efficiency. To solve issue, we propose generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, first design novel Token Slimming Module (TSM), can boost inference efficiency ViTs by dynamic token aggregation. As general method dropping, our TSM softly integrates redundant into fewer informative ones. It dynamically zoom visual attention without cutting off discriminative relations in images, even with high slimming ratio. Furthermore, introduce concise Feature Recalibration Distillation (FRD) framework, wherein reverse version (RTSM) recalibrate unstructured flexible auto-encoder manner. Due similar structure between teacher student, FRD effectively leverage knowledge better convergence. Finally, conduct extensive experiments evaluate demonstrates that speed up $$\mathbf {1.7}\times $$ negligible accuracy drop, {3.6}\times while maintaining $$\textbf{97}\%$$ their performance. Surprisingly, simply arming LV-ViT SiT, achieve new state-of-the-art performance ImageNet. Code is available at https://github.com/Sense-X/SiT .